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This article attempts to verify some assumptions evident in the scenario planning literature through the 
application of quantitative measures. The Dimensions of the Learning Organization Questionnaire is used 
to measure participant perceptions of the learning organization characteristics pre- and post- scenario 
planning intervention. Results are discussed, limitations are identified and clarified, and conclusions are 
drawn with speculations and refinements for future research. 
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Scenario planning has been reported as a method of creating, enhancing, and improving learning organization 
characteristics (Georgantzas & Acar, 1995; van der Heijden, 2004; Wack, 1985). While these claims have been 
abundant in the scenario planning literature, there has been no quantitative support for these claims. Efforts to 
evaluate scenario planning through quantitative analyses do not exist (Chermack, 2002; Wack, 1985) and for 
scholars interested in establishing and documenting the effects of scenario planning in an observable, consistent and 
careful manner, the current lack of sound scenario planning research provides a ripe agenda. 

While there is not yet evidence of a correlation between scenario planning, and the construct of organizational 
learning (which is considered a necessary component of the learning organization), there does seem to be some 
evidence of individual learning in the scenario planning experience although none of it is documented in a rigorous 
or careful manner. Many authors have based their work in cognitive psychology, and specifically, in the concept of 
mental models (Ringland, 2002; Senge, 1994; Shoemaker, 1992; van der Heijden, 1997). This work is also 
generally descriptive and conceptual and often refers to the classic Royal Dutch / Shell examples and cases (Wack, 
1985a, 1985b). These cases and descriptive examples have fueled larger claims that are simply unfounded 
according to general research methods, however, they do spark interest in investigating the phenomena more 
closely. To be clear, while individual and organizational learning seem to be and are reported as intricate 
components of successful scenario planning, their relationships must be examined more closely. 

This article does not intend to be overly critical of scenario planning pioneers. On the contrary, let the work of 
these exceptionally creative and generative thinkers be acknowledged (Ringland, 2002; Senge, 1994; Shoemaker, 
1992; van der Heijden, 1997; van der Heijden, Bradfield, Burt, Cairns & Wright, 2002; Wack, 1985a, 1985b). This 
article does however argue that the time has emerged for more careful documentation of what contributes to the 
success of scenario planning and what contributes to its failure. Furthermore, this careful documentation might best 
be conducted through the use of academic tools that have been neglected thus far in the analysis of scenario planning 
projects. 

Theoretical Framework 

Chermack (2004) devised a method of evaluating performance-based scenario planning. The method consisted of 
evaluating scenario planning in terms of economic, systems, and psychological domains at the individual, group, and 
organizational levels. Chermack (2003) developed an evaluation heuristic and also a research agenda for assessing 
performance based scenario planning according to multiple measures and from multiple methodological approaches 
(see Figure 1). This article reports on quantitative research that explores ONE aspect of that matrix — namely, 
psychological performance (learning) at the organizational level. To clarify, this article explores the effects of 
scenario planning on perceptions of learning at the organizational level. 

Copyright © 2006 Thomas J. Chermack, Susan A. Lynham and Louis van der Merwe 
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Figure 1. Heuristic for Evaluating Scenario Planning in Performance Contexts (Based on Swanson, 1994 and 
Rummler and Brache, 1995, from Chermack, 2003) 
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Scenario Planning 

Scenario planning is a multi-faceted organizational intervention aimed at recognizing the fact that the business 
environment is uncertain. Thus scenario planning assumes the best way to cope with uncertainty is to try and 
include it in the planning process. Based on both external and internal analyses, scenario planning usually occurs 
over multiple workshops in which key driving forces in the business environment are identified as well as the main 
concerns of managers and executives and is usually coupled with general strategic planning. A distilled list of issues 
is compiled and then ranked first according to impact on the business agenda and then uncertainty. This ranking 
exercise produces a matrix with impact on the “X” axis and uncertainty on the “Y” axis. Elements ranked high in 
both categories become the core elements of the scenarios themselves. Further workshops allow participants to 
explore the “what ifs” in each of the scenarios and eventually, the current strategic agenda is assessed in the context 
of each scenario. A general stepped approach to scenario planning according to the Centre for Innovative 
Leadership (1994) is as follows: 

1) Identify a strategic organizational agenda, including assumptions and concerns about strategic thinking 
and vision, 

2) Challenge existing assumptions of organizational decision makers by questioning current mental models 
about the external environment, 

3) Systematically examine the organizations external environment to improve understanding of the structure 
of key forces driving change, 

4) Synthesize information about possible future events into three or four alternative plots or story lines 
about possible futures, 

5) Develop narratives about the story lines to make the stories relevant and compelling to decision makers, 
and 

6) Use stories to help decision makers “re-view” their strategic thinking. 

Scenario planning is generally regarded as something of an organic process and has thus been viewed as difficult to 
study and there are many complicating factors. Certainly, creative elements in the scenario process must be allowed 
to emerge, and so the steps provided are merely a framework for structuring scenario projects. 

The Learning Organization 

Yang, Watkins and Marsick (2004) clarified: 

A difference between two related yet distinct constructs -- the learning organization and organizational learning. 
The construct of the learning organization normally refers to organizations that have displayed these continuous 
learning and adaptive characteristics, or have worked to instill them. Organizational learning, in contrast 
denotes collective learning experiences used to acquire knowledge and develop skills, (p. 34) 

The importance of this distinction for the purposes of this study is significant in that it provides the opportunity 
to suggest that all learning organizations feature learning at the individual, group and organizational levels. It is not 
feasible, however, to suggest that all organizations that engage in learning at the organizational level can be labeled 
as learning organizations. 

Thus, we arrive at the preliminary conclusion that the learning organization is much more than an organization 
that instills and promotes learning at the organizational level. More importantly, that organizational learning can be 
thought of as aggregate learning, or the sum of multiple individual learning experiences. For the purposes of this 
study, it is important to clarify that a learning organization is an organization thought to be capable of continuous 
learning and adaptation. While the scenario planning literature does not get to a level of detail sufficient to 
differentiate these terms, one additional feature of this article is that it provides clarification in line with the more 
developed literature on the learning organization. 
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Scenario planning literature proposes that organizational learning is a good measure of scenario planning based 
on the nature of scenario planning itself. That is, scenario planning purports to change the way participants view the 
world by expanding their abilities to consider possibilities. This expansion is purported to take place through an 
alteration of individual mental models. Cognitive theory espouses that learning is the primary driver of mental 
model change. 

Dimensions of the learning organization. Watkins and Marsick have developed the Dimensions of the Learning 
Organization Questionnaire based on over 30 years of experience working with organizations interested in 
increasing their capacity to support learning at all levels (1997; 2003). Their instrument is based on seven key 
constructs. These constructs are defined in Table 1. 

Table 1. Definitions of Constructs for the Dimensions of the Learning Organization Questionnaire (Watkins and 
Marsick, 2003, p. 139). 


Dimension 


Definition 


Create continuous learning opportunities 


Learning is designed into work so that people can learn on the job; 
opportunities are provided for ongoing education and growth 


Promote dialogue and inquiry 


People gain productive reasoning skills to express their views and 
the capacity to listen and inquire into the views of others; the 
culture is changed to support questioning, feedback, and 
experimentation 


Encourage collaboration and team learning 


Work is designed to use groups to access different modes of 
thinking; groups are expected to leam together and work together; 
collaboration is valued by the culture and rewarded 


Create systems to capture and share learning 


Both high- and low-technology systems to share learning are 
created and integrated with work; access is provided; systems are 
maintained 


Empower people toward a collective vision 


People are involved in setting, owning and implementing a joint 
vision; responsibility is distributed close to decision making so that 
people are motivated to learn toward what they are held accountable 
to do 


Connect the organization to its environment 


People are helped to see the effect of their work on the entire 
enterprise; people scan the environment and use information to 
adjust work practices; the organization is linked to its communities 


Provide strategic leadership for learning 


Leaders model, champion, and support learning; leadership uses 
learning strategically for business results 



Each of these constructs is measured by a collection of items on the instrument and each of these constructs is 
considered a characteristic of the learning organization (Argyris and Schon, 1996; Senge, 1990; Watkins and 
Marsick, 1996; 1997; 2003) measured by a six point Likert scale. The questions are directed at individual behaviors. 
Further details concerning the DLOQ instrument itself are presented in a discussion of the methodology. 

Having outlined the lack of research based studies in scenario planning and having reviewed the key concepts 
and purpose of this study, it is clear that organizational learning and learning organization literatures are 
conceptually related to scenario planning, and thus, the logic for undertaking this study is hopefully made clear. 
That is, this study seeks to explore organizational learning as defined by the seven constructs submitted by Watkins 
and Marsick (1996; 1997; 2003) as a reasonable and desirable outcome of scenario planning. 

Purpose of the Article and Research Question 

This article addresses two core issues with regard to scenario planning: 1) a lack of data-based and empirically 
driven studies about the phenomenon, and 2) a lack of general evaluation of scenario planning practices. The 
purpose of this article is to describe the relationship between scenario planning and perceptions of the learning 
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organization using quantitative methods. Thus, the research question that served as the basis of this inquiry was as 
follows: 

RQ1 — What are the effects of scenario planning on individual perceptions or the organizations’ ability to 
continuously learn and adapt? 

Hypothesis 

We hypothesize an increase in mean scores on the seven constructs of the learning organization as measured by 
the DLOQ. More specifically, we hypothesize that participation in scenario planning will change participant 
perceptions of their organizations’ capability to continuously learn and adapt. The null hypothesis and our 
hypothesis are provided symbolically as follows: 

H 0 : Md ~ 0 
H\ '■ Md — 0 

Method 

The following sections detail the research method, sampling strategy, and instrument used in the conduct of this 
research study. 

Sample 

Participants in a scenario planning project at a large educational institution in the southern United States served 
as the sample for this research study. Participants were selected on the basis of a single criterion — participation in 
the scenario planning project. Given the lack of data-based inquiry that examines the scenario planning phenomena, 
this study aimed primarily to provide a data-based assessment of scenario planning practices. In total there were ten 
participants (N = 10). One participant dropped out of the planning project leaving a total of 9 participants that 
completed the entire project and submitted data points. Clearly the sample size is a limitation and an analysis of this 
limitation is discussed in depth after the implications of the study are presented 
Instrument 

The learning organization is a popular, though ill-understood phenomenon. “The concept of the learning 
organization has received increasing attention in the field of organization studies, yet little is known about how to 
measure it" (Yang, Watkins & Marsick, 2004, p. 31). Most writing and research on the concept of the learning 
organization and organizational learning is conceptual and descriptive, offering little empirical evidence (Tsang, 
1997). 

The task of choosing an instrument, however, was simple. In 2004, Yang, Watkins and Marsick provided 
general evidence of reliability and validity for the Dimensions of Learning Organization Questionnaire developed by 
Watkins and Marsick in 1999. One of only a few instruments available that measures constructs of the learning 
organization, and the only one that has undergone extensive statistical measurements of reliability and validity, the 
DLOQ seemed a clear choice for the purposes of the study. 

The DLOQ features seven dimensions that are reflective of characteristics of learning organizations identified in 
several years of literature reviews and case studies and were presented in Table 1. These dimensions include 1) 
continuous learning, 2) dialogue and inquiry, 3) team learning, 4) embedded systems, 5) empowerment, 6) system 
connections and 7) leadership. 

Validity. The DLOQ has undergone significant validity testing. Yang, Watkins and Marsick (2004) employed 
criterion indices for assessing their model validity. These six indices included the chi-square test, goodness-of-fit 
index, goodness-of-fit index adjusted for degree of freedom, Bentler’s (1990) comparative fit index, Bentler and 
Bonnett’ s (1980) non-normed fit index and Steiger’s (1990) root mean square error of approximation. All of these 
tests yielded the conclusion that the instrument is a valid one. For full discussion, please see Yang, Watkins and 
Marsick (2004). Further, several additional studies (Ellinger, Ellinger, Yang and Flowton, 2003; Yang, 2003; 2004) 
have provided validity analyses that lend support to the claim that the DLOQ is a valid instrument. Due to our small 
sample size, we did not think it would be beneficial to perform these tests again, however, this reporting of previous 
testing is meant to provide a general sense that the DLOQ is an appropriate proxy for learning organization 
characteristics. 

Further studies are emerging (such as Ellinger, Ellinger, Yang & Flowton) that correlate the perception-based 
data in the DLOQ with more objective measures of organization performance (such as financial and stock market 
performance). These studies suggest that the DLOQ is indeed a good proxy for actual organization performance in 
general. Additional studies that link data from the instrument with objective measures of organizational 
performance (as in Ellinger, Ellinger, Yang & Flowton, 2002) will continue to increase the validity of the 
instrument. 

Data Collection and Analysis 
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The research design for this project included general descriptive statistics and a paired sample t-test of pre- 
scenario planning and post- scenario planning scores on Watkins and Marsick’s Dimensions of the Learning 
Organization Questionnaire. The t-test was chosen because it provides a measurement of mean scores based on 
inventories taken before and after the scenario planning intervention. The overall strategy for this study was to take 
a measurement with the Dimensions of the Learning Organization instrument prior to participation in scenario 
planning and then another measurement using the same instrument upon completion of the scenario planning 
project. Approximately three months time passed between measurements. The idea was then to compare the pre- 
and post- scores and attempt to attribute any significant differences in mean scores on the seven constructs of the 
DLOQ to participation in the scenario planning process. 

The objective of the data analysis was to examine claims that scenario planning is a method of enhancing or 
improving organizational learning, or developing a learning organization. Questionnaires were administered at 
meetings held at times scheduled for the scenario planning project. At least one of the authors was present at each 
session. Participants were given the questionnaires and asked to return them upon completion, which took 
approximately one hour for each meeting. The analysis began with examining simple descriptive statistics, 
specifically looking at participant variance among the seven dimensions (according to mean scores). Further 
analysis included paired sample t-tests. 

Results 

A reliability analysis of the DLOQ scale for this study using Cronbach’s Alpha returned a value of .9370. 
According to Nunnely (1970) a value of .70 or greater generally indicates a reliable instrument. General descriptive 
statistics for both pre and post data are provided in Table 2. 

Table 2. Descriptive Statistics for Participant Responses on the DLOQ 



Variable 


Description 


N 


Minimum 


Maximum 


Mean 


Sid. 

Deviation 


CONTLEAR 


Create continuous learning 
opportunities 


9 


2.71 


5.00 


4.00 


0.84 


DIALINQU 


Promote dialogue and inquiry 


9 


3.50 


5.00 


4.48 


0.70 


TEAMLEAR 


Encourage collaboration and 
team learning 


9 


3.00 


5.33 


4.38 


0.77 


EMBSYS 


Create systems to capture and 
share learning 


9 


1.00 


4.50 


3.37 


1.04 


EMPOWER 


Empower people toward a 
collective vision 


9 


3.83 


5.00 


4.25 


0.64 


SYSCONN 


Connect the organization to its 
environment 


9 


1.83 


4.67 


4.10 


1.09 


LEADERS 


Provide strategic leadership for 
learning 


9 


2.50 


5.33 


4.4 


0.86 


CONTLEAR2 


Create continuous learning 
opportunities 


9 


4.29 


5.57 


5.14 


0.43 


DIALINQU2 


Promote dialogue and inquiry 


9 


4.83 


5.83 


5.21 


0.33 


TEAMLEAR2 


Encourage collaboration and 
team learning 


9 


4.83 


5.67 


5.21 


0.29 


EMBSYS2 


Create systems to capture and 
share learning 


9 


2.67 


5.5 


4.13 


1.06 


EMPOWER2 


Empower people toward a 
collective vision 


9 


4 


5.67 


5.06 


0.61 


SYSCONN2 


Connect the organization to its 
environment 


9 


2.83 


5.83 


5 


0.98 


LEADERS2 


Provide strategic leadership for 


9 


4.83 


6 


5.29 


0.42 



learning 

Valid N Create continuous learning 9 

(listwise) opportunities 
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The constructs of greatest variance among the participant responses were “embedded systems” and “system 
connections”. This could be because these constructs are more abstract than the others in the instrument and may 
have the greatest range of possible interpretations by participants. Table 3 provides the results of the t-test. 



Table 3. Paired-Sample T-Test Results for the Seven Dimensions of the DLOQ 







Paired Differences 
Mean 


Std. Deviation 


t 


Sig. (1-tailed) 


Pair 1 


CONTLEAR - CONLEAR2 


1.21 


1.18 


2.913* 


.0115 


Pair 2 


DIALINQU - DIALINQ2 


.83 


.93 


2.528* 


.0195 


Pair 3 


TEAMLEAR - TEAMLEA2 


.90 


1.02 


2.499** 


.0070 


Pair 4 


EMBSYS - EMBSYS2 


.87 


1.88 


1.318 


.1145 


Pair 5 


EMPOWER - EMPOW2 


.82 


1.08 


2.134* 


.0350 


Pair 6 


SYSCONN - SYSCON2 


1.17 


1.30 


2.524* 


.0200 


Pair 7 


LEADERS - LEADERS2 


1.13 


.69 


4.585** 


.0015 



**=significant at a<.01 / *=significant at a<.05 



Discussion and Conclusions 

The scenario planning intervention is associated with increased perceptions of organizational learning across the 
board. Six of the seven constructs on the DLOQ showed a significant change from pre- to post- test scores, four 
were significant at the p = .05 level and two were significant at the p = .01 level. 

For the variable “Continuous Learning” a t-score of 2.913 was returned indicating a significant difference in 
pre- and post- scores. There have been considerable claims in the scenario planning literature that the process is 
based on an idea of continuous learning. Our data does seem to support this claim overall, however the limitations 
of a small sample size are unavoidable. “Dialogue and Inquiry” also increased (SD = .93) and was found significant 
with a t score of .0195. Kolb’s (1984) learning loop is frequently cited as a relevant model to represent how leading 
scenario thinkers suggest that learning takes place throughout the exercise. Van der Fleijden (2002) has also 
published theoretical work on the espoused purposes of scenario planning, one of which he claims is to engage the 
organization in continuous learning practices. There is also considerable support for the significant findings with 
regard to dialogue and inquiry. Dialogue is the primary means through which scenario planning occurs, as it is 
largely an exercise in communicating points of view. Georgantzas & Acar (1995), van der Fleijden (1997), 
Schwartz (1991), Wack, 1985), Ringland (1995) and Senge (1994) have linked scenario work to that of David Bohm 
and expressed that dialogue and inquiry are the primary means through which mental models are revealed and 
changed. The variable with the most significant increase in score was “Team Learning” (SD = 1.02; t = 2.50) which 
is not surprising given the close interpersonal and team learning requirements of scenario planning. Team learning 
is also a critical component argued in the scenario literature (Courtney, 2002; Godet, 2000; Senge, 1994). Because 
the people involved in the scenario process are so critical to its outcome, the scenario team should be assembled 
very carefully, often involving outside thinkers referred to as “remarkable people” (van der Fleijden, 1997). 
“Empowering People” was a significant construct ( t = 2.13) and some theoretical work has supported the 
assumption that scores on this construct would increase (Wack, 1985a). Specifically, scenario planning is thought to 
be the most provocative when teams of individuals from all levels of the organization are involved and are charged 
with creating significant inputs to the scenario planning process (Schwartz, 1991). “System Connections” was a 
significant construct ( t = 2.52) and this significance may demonstrate the close link between the kind of thinking 
required in scenario planning and popular system theory. Scenario work is clearly based in systems thinking as most 
any of the prevalent books and articles on scenario planning refer to the work of Senge and his considerable 
contributions to the systems literature in organizational contexts. It is also possible to see this link in the Fifth 
Discipline Fieldbook, which is published as a guide to organizational learning and contains numerous articles by 
leading scenario thinkers. Finally, “Leadership” was a highly significant construct ( t = 4.59) which may verify 
assumptions and anecdotes that scenario projects require top-level support or they usually don’t have much impact. 

The construct that did not return significant results was “Embedded Systems”. More specifically, the 
description of these constructs from Watkins and Marsick (2003) are as follows: “Both high- and low-technology 
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systems to share learning are created and integrated with work; access is provided; systems are maintained” (p. 139). 
There is minimal mention of this construct in the scenario planning literature. While scenario planning itself is 
viewed by some as a technology for sharing learning, few companies use scenario planning on an ongoing basis. 
Van der Heijden (2004) discussed four purposes of scenario planning — one of which was scenario planning as a 
continuous learning tool. He also suggested that few companies use scenarios for this purpose, but that it was 
possibly the use with the greatest potential impact on the organization. 

It is extremely important to note that while there are numerous writings that support the findings of this small 
study, none of these writings draw on empirically analyzed data. Some case studies have produced valuable insights 
into how the process of scenario planning works, but not in a way that allows us to generalize to the majority of 
scenario exercises. And so this study attempts to provide a first step in studying scenario planning in a manner that 
aims for this kind of generalization. Of course, there are some serious limitations that do not allow this study to 
attain a level consistent with such generalization. We provide our results and hope that they may lay the foundation 
for future studies that go beyond theoretical of conceptual contributions to the development of scenario planning 
practices. 

Limitations and Recommendations for Future Research 

While this study does provide a data based assessment of some portion of the relationship between scenario 
planning, organizational learning, and the potential for developing a learning organization, there are some 
considerable problems with the research presented. Dillman (2000) identified four sources of error in survey 
research, namely, 1) sampling error, 2) coverage error, 3) measurement error and 4) nonresponse error. Overall, this 
study is in violation of only one of these sources of error, but it is to a great extent. The sample size for this study is 
small enough that the results could simply be attributed to chance. That is, the sample size is not large enough to 
alleviate the possibility that the results could have simply become significant by chance. A small sample size limits 
the conclusions that can be drawn from the data and generalizability as a whole. Our data, however, supports claims 
made by previous literature reviews and conceptual articles, and is simply arguing for further development of this 
area. We do not attempt to draw broad conclusions about the effects of scenario planning interventions, rather we 
submit our findings as a first step toward investigating the effects of scenario planning on learning in organizations. 

While sample size is certainly a critical flaw in the study, a lack of research-based studies in the scenario 
planning literature could indicate that there would be some value in reporting the results that were found. In 
addition, there are clear implications for moving forward. First, while it is unlikely that most studies of scenario 
planning practices will include large samples (for example, samples of 200 or more), the solution may be to begin 
studying the phenomenon longitudinally. That is, small samples from similar situations over long periods of time 
could be combined into larger samples that would avoid the problems with error found in this study and others with 
small samples. 

It is also possible that large enough samples could come from the expanding contexts in which scenario 
planning is applied. For example, scenarios are being adopted as tools for exploring the possible futures of small 
countries and developing nations. In these contexts (and possibly large enough corporate contexts) situations could 
arise that involve enough participants to alleviate problems with sample size. In any event, further research must be 
done with proper sample sizes to investigate claims that scenario planning improves learning in organizations and is 
a tool for developing learning organizations. The design of this study is sound, but could be improved with the use 
of a control group. Therefore, further research that we intend to undertake will involve these adjustments. 

Contribution to New Knowledge in HRD 

It is our view that strategy-making is inherently a process related to human resources. Like all organizational 
processes, people ultimately create and implement the strategy. Torraco and Swanson proposed two general 
strategic roles of HRD in 1995 as supporting and shaping organization strategy. Little has been done within the 
academic field of HRD to expand upon these roles, although practicing HRD managers vary tremendously in their 
involvement in strategy -- some clearly in positions to shape and support organization goals and intent. We have 
committed to exploring how HRD professionals might best contribute to shaping and supporting organization 
strategy and we believe strongly that scenario planning is the vehicle that links the work of HRD professionals to the 
strategic context of organizations. Research such as that described in this article (but more effectively, the research 
that this article might lay the foundation for) can provide the path for HRD professionals to understand and then 
perform scenario planning in organizations, thus fulfilling a role that both shapes and supports organization strategy. 
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